Troubleshooting
Common failure modes during and after Yeedu CFT deployment, with diagnosis steps and fixes.
Lambda Timeout / Stack Rollback
Symptom: Stack rolls back at YeeduSetupLambdaInvoke with status CREATE_FAILED. CloudFormation events show:
Custom Resource failed to stabilize in expected time
or
Received FAILED signal with UniqueId ...
Diagnosis:
# Tail Lambda logs (replace yeedu-prod with your stack name)
aws logs tail /aws/lambda/yeedu-prod-yeedu-setup --since 2h --follow
Common causes and fixes:
| Cause | Fix |
|---|---|
s3:GetObject denied on yeedu-softwares | Verify the Lambda execution role has s3:GetObject on arn:aws:s3:::yeedu-softwares/releases/*. Check for SCP denials in your AWS Organization. |
| Target S3 bucket access denied | Lambda validates bucket access first (head_bucket). Verify the bucket exists and Lambda role has s3:ListBucket on it. |
ECR CreateRepository denied | Lambda role must have ecr:CreateRepository on arn:aws:ecr:<region>:<account>:repository/yeedu_*. |
Core-Services-Formation.tar.gz not found in yeedu-softwares | The iYeeduVersion value may not exist in the Yeedu S3 bucket. Contact Yeedu support to confirm the version string. |
| Lambda timeout (900s exceeded) | The S3 object being copied may exceed Lambda's network throughput capacity. This is rare for artifact files — contact Yeedu support. |
EC2 UserData Failure
Symptom: EC2 creation fails with:
WaitCondition received a failure signal with UniqueId ...
or the CreationPolicy timeout (60 min) expires.
Diagnosis:
The bootstrap log is streamed to CloudWatch automatically. Check it first:
# Get EC2 instance ID
INSTANCE_ID=$(aws cloudformation describe-stack-resources \
--stack-name yeedu-prod \
--query "StackResources[?LogicalResourceId=='EC2Instance'].PhysicalResourceId" \
--output text)
# Tail the bootstrap log
aws logs tail yeedu_userdata_logs \
--log-stream-name $INSTANCE_ID \
--since 2h --follow
Alternatively, if the instance is still running, SSM in:
aws ssm start-session --target $INSTANCE_ID
sudo tail -f /var/log/bootstrap.log
Common causes and fixes:
| Symptom in log | Cause | Fix |
|---|---|---|
mount: Connection timed out | EFS mount failure | Verify EFS SG allows TCP 2049 from EC2 SG. Check EFS mount targets exist in correct subnets. |
AccessDeniedException on GetSecretValue | Secrets Manager access denied | EC2 IAM role is scoped to yeedu* secrets. Verify stack name matches (secrets are named <stack-name>-...). |
ERROR: Core-Services-Formation.tar.gz not found | S3 artifact missing | Lambda should have copied it. Check Lambda logs to confirm prepare_yeedu_software completed. Verify s3://<bucket>/yeedu/Core-Services-Formation.tar.gz exists. |
ERROR: cfn-signal not found in PATH | cfn-bootstrap install failed | Check apt-get and python3 setup.py install output earlier in the log. May be a transient apt mirror issue — try redeploying. |
Cannot connect to Docker daemon | Docker not started | Check systemctl status docker on the instance. docker.io install may have failed. |
Docker image docker load fails | Large .tar file, disk full | EC2 has 200 GB root volume. Check df -h. Also verify all .tar files downloaded correctly from S3. |
yeedu-core-services.sh not found | Extraction failed | Check that tar -xzf Core-Services-Formation.tar.gz completed without errors. |
ALB 502 Bad Gateway
Symptom: Navigating to the Yeedu UI (https://<alb>:8012 or http://<alb>:8002) returns 502 Bad Gateway.
Diagnosis steps:
-
Check target health:
TG_ARN=$(aws cloudformation describe-stack-resources \
--stack-name yeedu-prod \
--query "StackResources[?LogicalResourceId=='TargetGroupUI'].PhysicalResourceId" \
--output text)
aws elbv2 describe-target-health --target-group-arn $TG_ARN -
Check containers on EC2:
aws ssm start-session --target <EC2InstanceId>
sudo docker ps
Common causes and fixes:
| Cause | Fix |
|---|---|
| Services still starting (first boot) | Docker images are large. Allow 10–15 minutes after EC2 becomes healthy for all containers to start. |
| EC2 security group not allowing ALB | Confirm EC2 SG has inbound TCP from ALBSecurityGroup on ports 8002, 8080, and 8081. |
| Wrong target group port | UI target group forwards to EC2 port 8002 (not 80). Verify the target group configuration. |
| Nginx/UI container not running | Check sudo docker logs <ui_container_name> --tail 50 |
Docker Image Push Failures (EC2 UserData)
Symptom: Bootstrap log shows errors during the docker push phase.
# Check ECR push errors in bootstrap log
grep -i "error\|failed\|denied" /var/log/bootstrap.log | grep -i ecr
Common causes and fixes:
| Cause | Fix |
|---|---|
| ECR login expired | The ECR auth token is valid for 12 hours. If UserData runs longer than 12 hours (unusual), re-login: aws ecr get-login-password | docker login ... |
| Repository doesn't exist | Lambda should have created all 17 repos. Check Lambda logs. If missing, create manually: aws ecr create-repository --repository-name yeedu_<name> |
Disk full during docker load | EC2 has 200 GB. Run df -h and docker system df. Prune unused images: docker image prune -f |
denied: User is not authorized | EC2 IAM role ECR permissions are scoped to arn:aws:ecr:<region>:<account>:repository/yeedu*. All Yeedu repo names start with yeedu_ — verify no typo in image name. |
RDS Connection Failures
Symptom: Yeedu services start but show database errors, or yeedu-connection.properties shows wrong endpoint.
Diagnosis:
# Test from EC2 directly
source /etc/yeedu-env
psql -h $YEEDU_RDS_ENDPOINT -p $YEEDU_RDS_PORT -U yeedu -d yeedu -c "SELECT 1"
Common causes and fixes:
| Cause | Fix |
|---|---|
fRDSEngineVersion not available in region | Find available versions and update. See command below. |
RDS not in ServiceSecurityGroup | Verify in the RDS console that the instance uses <stack-name>-service-sg. |
EC2 not in ServiceSecurityGroup | EC2 must be in both EC2SecurityGroup and ServiceSecurityGroup. Check attached SGs in EC2 console. |
| Wrong RDS secret values | The YeeduRdsSecret is populated with YeeduDb.Endpoint.Address at deploy time. Check current values: aws secretsmanager get-secret-value --secret-id yeedu-prod-prod-rds-secret |
# Find available PostgreSQL 15.x versions in your region
aws rds describe-db-engine-versions \
--engine postgres \
--query "DBEngineVersions[?starts_with(EngineVersion,'15')].EngineVersion" \
--output table
Security Group Quick Reference
ALB Security Group — inbound from internet
| Port | Protocol | Source | Purpose |
|---|---|---|---|
| 8012 | TCP | 0.0.0.0/0 | HTTPS — Yeedu UI (SSL termination at ALB) |
| 8002 | TCP | 0.0.0.0/0 | HTTP — Yeedu UI (non-SSL or EC2 target) |
| 8080 | TCP | 0.0.0.0/0 | REST API |
| 8081 | TCP | 0.0.0.0/0 | Functions Proxy |
EC2 Security Group — inbound from ALB and YeeduInternal SG
| Port | Protocol | Source | Purpose |
|---|---|---|---|
| 8002 | TCP | ALB SG | Yeedu UI |
| 8080 | TCP | ALB SG | REST API |
| 8081 | TCP | ALB SG | Functions Proxy |
| 8888–9088 | TCP | YeeduInternal SG | Jupyter workspace ports |
| 8003 | TCP | YeeduInternal SG | RabbitMQ management UI |
| 8004 | TCP | YeeduInternal SG | Redis |
| 8005 | TCP | YeeduInternal SG | RabbitMQ AMQP |
| 8006 | TCP | YeeduInternal SG | PostgreSQL |
| 8007 | TCP | YeeduInternal SG | Spark History Server |
| 8008 | TCP | YeeduInternal SG | Grafana |
| 8086 | TCP | YeeduInternal SG | InfluxDB |
| 8009 | TCP | YeeduInternal SG | LDAP |
| 8010 | TCP | YeeduInternal SG | Yeedu UI (internal) |
| 8200 | TCP | YeeduInternal SG | HashiCorp Vault API |
| 8011 | TCP | YeeduInternal SG | Functions service |
| 8765 | TCP | YeeduInternal SG | Proxy cacher |
| 22 | TCP | 0.0.0.0/0 | SSH — restrict to your CIDR in production |
EFS Security Group — inbound NFS
| Port | Protocol | Source | Purpose |
|---|---|---|---|
| 2049 | TCP | EC2 SG | NFS from main Yeedu EC2 |
| 2049 | TCP | YeeduInternal SG | NFS from Spark worker EC2s |